Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Markzipan · 2025-10-03T08:23:18Z

Glues FES into build_runner, which is our first step towards hot reload + build_runner.
The high level workflow is:

A persistently running frontend server is initialized once when a build starts.
build_runner requests JS files based on locally modified/generated dart files (as usual). Builders that collect meta-information about DDC modules also - as a side effect - record the main app entrypoint and any locally modified files.
When a JS file is requested, the frontend server receives recompile requests via a proxy queue (to maintain communication order with the frontend server).
The frontend server processes compilation requests and serves compiled JS files back to build_runner (hot-reload ready).

Major changes:

Adds a DdcFrontendServerBuilder to our set of DDC builders (enabled via the web-hot-reload config). This builder keeps a PersistentFrontendServer instance alive across rebuilds. Compile/recompile requests are queued via a FrontendServerProxyDriver resource.
Uses scratch_space to record both 1) the main app entrypoint and 2) updated local files from the entrypoint_marker builder and the module_builder builder respectively. These are side effects that break certain stateful 'guarantees' of standard build_runner execution. The entrypoint_marker builder runs before any of the downstream DDC builders and finds the web entrypoint, as Frontend Server must receive the same entrypoint on every compilation request.
Requires that strongly connected components in both the frontend server and build_runner be disabled.

Test changes:

Extends build_test to permit incremental builds. This involves passing the asset graph + asset reader/writer across build results and only performing cleanup operations after a series of rebuilds.
build_test doesn't support runs_before and other ordering rules in build.yaml, so the above changes allows a kind of imperative ordering, which is important for testing entrypoint_marker.

Minor changes:

Added a flag to disable strongly connected components in build_web_compilers (implemented using raw ddc meta-modules over clean ddc meta-modules + enforcing fine module aggregation).
Added disposal logic to scratch_space so that rebuilds only retain modified files.
Updated scratch_space package_config.json specs (packageUri and rootUri). The previous values didn't seem to make sense to me, but I'm also not familiar with how that's standardized in scratch_space.
Added file and uuid deps to build_modules.
Moved around some helper functions.
Ported some naming functions from the DDC runtime.

Currently doesn't support live-reloading (functionality appears to have been broken a while ago). This'll be added in an upcoming change and permit webdev-like auto-hot-reload on save (on top of manual).

Enable this by adding the following to a project's build.yaml:

global_options:
  build_web_compilers|sdk_js:
    options:
      web-hot-reload: true
  build_web_compilers|entrypoint:
    options:
      web-hot-reload: true
  build_web_compilers|ddc:
    options:
      web-hot-reload: true
  build_web_compilers|ddc_modules:
    options:
      web-hot-reload: true

…r fine modules for hot reload

github-actions · 2025-10-03T08:25:28Z

PR Health

Changelog Entry ❗

Package	Changed Files
package:build_modules	build_modules/lib/build_modules.dart build_modules/lib/src/common.dart build_modules/lib/src/ddc_names.dart build_modules/lib/src/frontend_server_driver.dart build_modules/lib/src/kernel_builder.dart build_modules/lib/src/module_builder.dart build_modules/lib/src/modules.dart build_modules/lib/src/scratch_space.dart build_modules/lib/src/workers.dart build_modules/pubspec.yaml
package:build_runner	build_runner/lib/src/constants.dart
package:build_test	build_test/lib/src/internal_test_reader_writer.dart build_test/lib/src/test_builder.dart build_test/lib/src/test_reader_writer.dart
package:build_web_compilers	build_web_compilers/lib/build_web_compilers.dart build_web_compilers/lib/builders.dart build_web_compilers/lib/src/common.dart build_web_compilers/lib/src/ddc_frontend_server_builder.dart build_web_compilers/lib/src/dev_compiler_bootstrap.dart build_web_compilers/lib/src/dev_compiler_builder.dart build_web_compilers/lib/src/sdk_js_compile_builder.dart build_web_compilers/lib/src/sdk_js_copy_builder.dart build_web_compilers/lib/src/web_entrypoint_builder.dart build_web_compilers/lib/src/web_entrypoint_marker_builder.dart build_web_compilers/pubspec.yaml
package:scratch_space	scratch_space/lib/src/scratch_space.dart

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

davidmorgan · 2025-10-03T14:35:20Z

Thanks!

This will take me a little work to digest :) I guess I can get to it in 1-2 days.

It definitely makes sense to expand support for testing. The details are hard :) ... in particular, I would like to avoid exposing any build_runner internals, including AssetGraph or how it's detecting file changes for follow-on builds.

So I will see if I can come up with a slightly differently suggestion.

Currently testBuilder uses fake build configuration and that's quite awkward; it starts to be weird when you talk about follow-on builds because there is no clear line for what should stay the same in the fake configuration and what's allowed to change.

One possibility would be to go all the way to real builds, I recently added a new type of integration test that I'm pretty happy with

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 17e38a6

I'll give it some thought :)

biggs0125 · 2025-10-03T17:21:06Z

Requires that strongly connected components in both the frontend server and build_runner be disabled.
Can you say more about the affect this has on the structure of compilations? Are we doing more work by turning off SCCs? My assumption was that SCCs allowed us to limit the scope of an invalidation.

Markzipan · 2025-10-03T17:53:18Z

@davidmorgan Thanks! It's a huge change, so please take your time. I wasn't able to configure it in a prettier way since I'm flying out for two weeks.

I'm highly unopinionated wrt testing, but it was definitely a struggle to get testBuilders to cooperate with rebuilds. Please send any followup recs for better ways to represent this test. The important bits to me are: 1) respecting builder order (as per build.yaml) since that matters now, 2) being able to run rebuilds vs new builds (or determine build boundaries), 3) updating files passed to the builders.

@biggs0125 The SCCs here operate on a the import-graph level. DDC's old module system requires that cyclic dependencies be 'unified' into a single module. Both the Frontend Server and build_runner doing this non-deterministically might cause a <--> b to be merged into a in build_runner and b in the Frontend Server, which gives us "import not found" errors.

With SCCs enabled, a.dart would invoke a single builder on module(a, b). Disabling them, two builders, module(a) and module(b), are invoked. The net effect of this on some apps I've tested hasn't been large, as most modules don't end up in an SCC anyway. The largest ramifications would be a hypothetical gigantic app that wants to bundle smaller JS files. But that would ideally be specified statically, so we'd be able to pass it consistently to both build_runner and the Frontend Server. The particularly annoying bit about SCCs is that both systems maintain their own independent algorithm without any configurability. I alternatively could've 1) poked a hole into the Frontend Server to allow per-compile module-to-lib maps and 2) had build_runner serialize its module-to-lib maps and send that per-compile to the Frontend Server. But I figured that was too much of a mess.

Re: failures. I think some tests are failing due to API changes without pinning new versions? I'll add those (maybe in a separate PR) if the current approach looks sound.

davidmorgan

This generally looks good, a few comments re: scratch_space.

Re: testing :)

You sort of don't need to pass assetGraph, it's serialized to the readerWriter and the next build will read it if it's there.

But, the next build will discard the asset graph if any builders changed, so you have to pass the exact same builders each time, which means you will get a full build the first time you call it anyway.

You commented with build.yaml ordering is not respected, and that's true. Adding workarounds for that is not too pretty, it's getting very close to a real build but now with quite a lot of configuration that doesn't exactly match a real build.

Would you be up for trying a different way of testing that is a real build?

I added tests recently that make it easy to set up some packages, run build_runner for real and check the output. Since you want persistent processes, watch mode would make sense, so something like:

build/build_runner/test/integration_tests/watch_command_test.dart

Line 1 in 9f07c7e

There is an example web build using the same test infra here

https://github.com/dart-lang/build/blob/master/build_runner/test/integration_tests/web_compilers_test.dart

The test infra is currently internal to build_runner, mainly build_runner/test/common/build_runner_tester.dart, if it looks useful let's just hack around that for now and I'll work out how best to clean it up, I guess adding to build_test is one option.

davidmorgan · 2025-10-06T10:06:38Z

build_runner/lib/src/constants.dart

 String get generatedOutputDirectory => '$cacheDirectoryPath/generated';

+/// Relative path to the cache directory from the root package dir.
+const cacheDir = '.dart_tool/build';


I don't see where this is used, remove?

davidmorgan · 2025-10-06T10:19:15Z

scratch_space/lib/src/scratch_space.dart

+  ///
+  /// This must be set before any asset builders run when compiling with DDC and
+  /// hot reload.
+  late AssetId entrypointAssetId;


This doesn't look particularly related to ScratchSpace, which I see is used by a few other builders.

I guess the problem here is of passing configuration between builders?

Maybe a new Resource class for that? It could also go in scratch_space.

davidmorgan · 2025-10-06T10:21:53Z

scratch_space/lib/src/scratch_space.dart

  final Directory tempDir;

+  /// Holds all files that have been locally modified in this build.
+  final changedFilesInBuild = <AssetId>{};


Can this be private and expose an Iterable? Maybe a read-only view if Set is needed?

To avoid any possibility of corrupted shared state if a builder thinks it's fun to write to it :)

davidmorgan · 2025-10-06T10:50:22Z

scratch_space/lib/src/scratch_space.dart

  /// Copies [assetIds] to [tempDir] if they don't exist, using [reader] to
  /// read assets and mark dependencies.
  ///
+  /// Locally updated assets will be recorded in [changedFilesInBuild].


I suggest making this and the comment for changedFilesInBuild more precise, e.g. here

Assets that have changed since the last time they were seen by ensureAssets are added to changedFilesInBuild.

And for changedFilesInBuild, maybe: Assets that changed between calls to ensureAssets. Cleared at the end of every build.

davidmorgan · 2025-10-06T10:51:24Z

scratch_space/lib/src/scratch_space.dart

@@ -1 +1 @@
 // Copyright (c) 2017, the Dart project authors.  Please see the AUTHORS file


I think scratch_space needs a new version number and then anything that needs what you're adding here should get a min dep onto that version?

Markzipan added 25 commits August 28, 2025 00:33

Initial support for DDC + FES with build_runner

cd52372

Adding hot reload flag consistency checks and enforcing module builde…

af089bf

…r fine modules for hot reload

Moving sources to generated scratch dirs

01e2411

Merging

414fc95

Only recompiling and invalidating requested source files

fa6594f

Merge branch 'master' of github.com:dart-lang/build

09a0486

Sample change with a hacky shared repo + digests impl comparisons

efec641

Moving shared state to scratch space

aae24a2

Merging

7aba038

addressing analysis errors

51f1e0d

additional file cleanup

5d309ca

Adding build_modules tests

9a37369

Adding build_modules tests

b2bab91

Merging

cba3f6b

Merge branch 'master' of github.com:dart-lang/build

244dcd4

Adding reset to frontend server driver

12a8af7

Adding a builder for labeling the app entrypoint.

b9e30b3

Removing the entrypoint from builder options

309b11f

Fixing error handling

625a922

Fixing some shutdown race conditions

150a42a

Adding reader-writer resets

c48e015

Adding support for incremental build testing.

9349807

Adding DDC frontend server builder tests

f108cb6

Updating entrypoint marker builder

75f3061

Merge branch 'master' of github.com:dart-lang/build

e248938

Markzipan requested review from jakemac53, biggs0125, nshahan and davidmorgan October 3, 2025 08:23

davidmorgan reviewed Oct 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Uh oh!

Markzipan commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

Uh oh!

davidmorgan commented Oct 3, 2025

Uh oh!

biggs0125 commented Oct 3, 2025

Uh oh!

Markzipan commented Oct 3, 2025 •

edited

Loading

Uh oh!

davidmorgan left a comment

Uh oh!

davidmorgan Oct 6, 2025

Uh oh!

davidmorgan Oct 6, 2025

Uh oh!

davidmorgan Oct 6, 2025

Uh oh!

davidmorgan Oct 6, 2025

Uh oh!

davidmorgan Oct 6, 2025

Uh oh!

Uh oh!

		@@ -1 +1 @@
		// Copyright (c) 2017, the Dart project authors. Please see the AUTHORS file

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Are you sure you want to change the base?

Adding incremental Frontend Server-compatible DDC builders + supporting infrastructure #4240

Uh oh!

Conversation

Markzipan commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

PR Health

Uh oh!

davidmorgan commented Oct 3, 2025

Uh oh!

biggs0125 commented Oct 3, 2025

Uh oh!

Markzipan commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidmorgan left a comment

Choose a reason for hiding this comment

Uh oh!

davidmorgan Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

davidmorgan Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

davidmorgan Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

davidmorgan Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

davidmorgan Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Markzipan commented Oct 3, 2025 •

edited

Loading